{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 正则化\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 零维正则化\n",
    "$$\n",
    "r = \\sum_i^m \\epsilon(\\theta_i)\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 一维正则化\n",
    "$$\n",
    "r = \\sum_i^m \\vert \\theta_i\\vert\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 二维正则化\n",
    "$$\n",
    "r = \\sum_i^m \\theta_i^2\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 多维正则化\n",
    "\n",
    "$$\n",
    "\\begin{aligned}\n",
    "r =& \\Vert \\theta \\Vert_p \\\\ = &\n",
    "\\sqrt[p]{\\sum_i^m {\\theta_i^p}}\n",
    "\\end{aligned}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "多维正则化也就是范数，不过常用的就是2范数，也就是<br>\n",
    "$$\n",
    "l_2 = \\sqrt{\\sum_i^m \\theta_i^2}\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 过拟合"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "模型在训练集上拟合数据，在测试集合上验证模型。<br>\n",
    "在强调拟合精度的训练集上，如果过于准确，而在测试集或者其他数据集上面缺乏泛化能力，这就称为过拟合。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "因此，模型训练中一般推荐使用简单模型进行训练，不仅直观，也是为了防止过拟合。<br>\n",
    "训练精度或者不高，但是泛化能力不差。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 正则惩罚\n",
    "## 基本形式\n",
    "$$\n",
    "J_L(\\theta) = J(\\theta) + \\lambda L\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "没有正则项之前，我们只是寻找到了$J(\\theta)$的一组最适用的$\\theta$<br>\n",
    "加上正则项之后，我们对于$\\theta$也有了要求。<br>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## L1\n",
    "一般采用$L_1$，具有稀疏性，因为是线性关系，所以一般交点在坐标轴上，会使得其他$\\theta_i$为$0$<br>\n",
    "经常用$L_1$来衡量不同$\\theta$之间的影响程度。<br>\n",
    "就泛化能力而言，很强大，但是有些偏执，而且拐点处不可导。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## L2\n",
    "和等高线相切，没有筛选能力，但是在交点处可导，一般使用$L_2$。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 过拟合原理"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "一般来说，训练出来的$\\theta$都会比较小，这是由于要衰减样本的影响，得到损失函数$J(\\theta)$的极小值。<br>\n",
    "但是如果$\\theta$不是最小，却得到极小的$J(\\theta)$，可能就是对样本数据的依赖，在特殊样本下得到的极小值，也就是过拟合。<br>\n",
    "为了泛化这种能力，必须均衡每个$\\theta$的影响能力，防止新数据集中单一对$\\theta$参数的放大，而得到异常的数据。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 正则化损失函数\n",
    "我们采用线性回归来观察一下\n",
    "$$\n",
    "\\begin{aligned}\n",
    "J_L(\\theta) =& \\frac{1}{2m}\\sum_i^m{\\left[  (h_\\theta(x_i) - y_i)^2 + \\lambda\\sum_i^n \\theta_i^2\\right]}\n",
    "\\end{aligned}\n",
    "$$\n",
    "\n",
    "$$\n",
    "\\theta_{next} = \\theta_j(1 - \\alpha \\frac{\\lambda}{m}) - \\frac{\\alpha}{m}\\sum_i^m(h_\\theta(x_i) - y_i)x_i\n",
    "$$"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "可以看到，在移动步率不变的情况下，每次的基准都是$\\theta_j$的一个比例，也就缩减了原来的$\\theta$，让它不会变化那么大。<br>\n",
    "让$\\theta$的各个参数尽量小的情况下尽量趋于中庸。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
